There exist numerous applications in which real time data analysis may be required. For example, data events may be collected in a financial setting to identify potentially fraudulent activity, in a network setting to track network usage, in a business setting to identify business opportunities or problems, etc. Often, it may be necessary to examine individual data events as they occur to immediately investigate any suspect behavior. Challenges however arise when analyzing data events in real time since historical data values are typically necessary to identify trends and patterns. Namely, accessing historical data can be a relatively slow process, and thus limits real time processing.
There exist various known techniques (e.g., running estimates, moving windows, etc.) for analyzing data events in real time (or near real time). Such techniques utilize little or no historical data to provide a statistical analysis of detected event values. Instead, they, e.g., maintain a running value, which is updated each time a new data event value occurs.
Because real time analysis techniques do not have the luxury of examining significant amounts of historical data, the efficacy of their results may be less than desirable, particularly where regular trends and patterns cannot be readily identified. For example, a given data event stream may be subject to a certain behavior (e.g., very high values) during one time period, and then subject to a different behavior (e.g., very low values) during a second time period, and then revert back to the first behavior during a third time period, etc. This type of “shifting” behavior significantly impacts the ability to analyze individual data events.
Accordingly, a need exists for a system and method of providing real time data analysis for data events subject to shifting behaviors.